Hi and welcome to our paper, VET, Visual Error Tomography for Point Cloud Completion and
High Quality Neural Rendering.
In our point-based novel view synthesis approach, we take a couple of dozen captured images
and a point cloud to create photorealistic neural renderings.
However, if the multi-view stereo or lidar point cloud is sparse or is missing geometry,
rendering quality is impacted heavily.
We fix this by introducing the concept of visual error tomography.
We create a 3D volumetric error representation based on visual error and spawn new points
where error densities are high.
To get this error volume, we collect the 2D error images obtained by comparing the result
of the blending point renderer to the ground truth and lift them to a 3D error volume via
CT reconstruction.
Sampling this error volume allows us to efficiently complete point clouds regardless of hole sizes
in the initial reconstruction.
As seen here, even fine details are reconstructed well, allowing our approach to generate competitive
renderings compared to established approaches.
In addition, our renderer optimizes point opacity, thus efficiently removing outliers
during training.
This is seen here with the water points incorrectly placed by callmap.
Our pipeline produces convincing results in various scenes.
Compared to the established one pixel point rendering approach ADOP, our blending renderer
shows improved temporal stability.
Even though not explicitly modeled, our pipeline handles view dependency well.
Thank you for watching.
Presenters
Zugänglich über
Offener Zugang
Dauer
00:03:37 Min
Aufnahmedatum
2023-11-13
Hochgeladen am
2023-11-13 16:06:07
Sprache
en-US
In the last few years, deep neural networks opened the doors for big advances in novel view synthesis. Many of these approaches are based on a (coarse) proxy geometry obtained by structure from motion algorithms. Small deficiencies in this proxy can be fixed by neural rendering, but larger holes or missing parts, as they commonly appear for thin structures or for glossy regions, still lead to distracting artifacts and temporal instability. In this paper, we present a novel neural-rendering-based approach to detect and fix such deficiencies. As a proxy, we use a point cloud, which allows us to easily remove outlier geometry and to fill in missing geometry without complicated topological operations.
Keys to our approach are (i) a differentiable, blending point-based renderer that can blend out redundant points, as well as (ii) the concept of Visual Error Tomography (VET), which allows us to lift 2D error maps to identify 3D-regions lacking geometry and to spawn novel points accordingly. Furthermore, (iii) by adding points as nested environment maps, our approach allows us to generate high-quality renderings of the surroundings in the same pipeline. In our results, we show that our approach can improve the quality of a point cloud obtained by structure from motion and thus increase novel view synthesis quality significantly. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively. Rendering quality outperforms state-of-the-art methods and temporal stability is significantly improved, while rendering is possible at real-time frame rates.